Skip to content

slng: rewrite plugin around the Unmute Bridge (v2)#6442

Merged
tinalenguyen merged 21 commits into
livekit:mainfrom
metehan-slng:slng-plugin-2.0
Jul 23, 2026
Merged

slng: rewrite plugin around the Unmute Bridge (v2)#6442
tinalenguyen merged 21 commits into
livekit:mainfrom
metehan-slng:slng-plugin-2.0

Conversation

@metehan-slng

@metehan-slng metehan-slng commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Breaking change

This is a major rewrite of livekit-plugins-slng. We would like it released as 2.0.0 (please bump accordingly at release time; version.py is left untouched per contribution guidelines).

What changed

The plugin now connects exclusively through SLNG's Unmute Bridge, the gateway's normalized protocol layer. A model identifier is all that is needed (slng.STT(model="deepgram/nova:3")); the plugin builds the bridge WebSocket endpoint itself.

  • Bridge-only routing. The 1.x model_endpoint / model_endpoints parameters were removed and now raise a clear ValueError pointing at the replacement (model= or connections=[...]). STT no longer defaults to a model; a model or connections must be passed explicitly.
  • Failover chains for STT and TTS. connections=[...] accepts model identifiers, bridge endpoint URLs, or typed STTConnectionConfig / TTSConnectionConfig objects. STT fails over at safe stream boundaries and replays buffered audio (including a pending finalize) onto the new connection; TTS fails over only before first audio. HTTP 413 is treated as terminal instead of walking the chain, since every candidate would reject the same oversized payload. Recovery back to the primary is cooldown-based.
  • Single-candidate resilience. With no further fallback candidate (the common single-model setup), a transient mid-utterance connection drop or a stalled-finalize watchdog timeout reconnects the same endpoint and replays the buffered audio rather than ending the stream, bounded by max_retry and reset on real transcript progress, so a persistently failing endpoint still surfaces the error.
  • Faster end-of-utterance. The stream sends the bridge a finalize signal when the agent's user state transitions to listening; wire it with stt.attach_to_session(session) (or call stt.notify_user_state(...) from a user_state_changed handler). A watchdog (final_timeout_s, opt-in) guards against providers that never deliver the final transcript. Interim results no longer disarm the watchdog.
  • BYOK support. provider_api_key forwards the customer's own provider credential as the gateway's X-Slng-Provider-Key header.
  • Geographic zone routing. In addition to the existing region_override, a new world_part_override constrains routing to a broad geographic zone (maps to the gateway's X-World-Part-Override header; region_override takes precedence). Both propagate to fallback candidates.
  • Typed plugin events. slng_event emits structured events for gateway session IDs and fallback attempt / switch / exhaustion.
  • Removed implicit behavior. Client-side language and voice normalization are gone by design: values are passed verbatim so they always match the selected model's contract (e.g. Sarvam requires BCP-47 hi-IN). Voice IDs are provider-native.
  • Honest capabilities. Only pcm_s16le STT input is accepted (previously other encodings were silently mislabeled), recognize() raises NotImplementedError (the bridge is WebSocket-only), and offline_recognize is reported as False.
  • Hardening. Bounded WebSocket receives with the connection timeout (mirroring the Cartesia plugin), lazy-starting fallback streams that support plain async for and collect(), per-run stream state so base-class retries produce audio, consistent watchdog and speech-event state across options-change reconnects, runtime voice/language/speed updates that propagate to fallback candidates and safely invalidate warm-standby connections, a bounded cap on silent reconnects, and no leaked connection or timing state.

Testing

  • make check passes: ruff format, ruff lint, and strict mypy via scripts/check_types.py.
  • The plugin is validated by an out-of-tree unit suite (scripted fake WebSockets, no network) covering failover with audio + finalize replay, single-candidate reconnect-and-replay on drop and on timeout, options-change reconnect state and event brackets, watchdog behavior, retry-safe stream construction, warm-standby invalidation, fallback stream contracts, phrase batching, geo override propagation, option forwarding, and encoding validation. Test files are not included in this PR (removed on request to keep it focused).
  • Verified live against the production SLNG gateway: STT flush-to-final latency, TTS synthesis across ElevenLabs / Cartesia / Deepgram / Rime bridge models, failover chains, BYOK, and world-part override steering (confirmed via gateway routing logs).

Notes for reviewers

  • README.md was rewritten to document the 2.0 API, including a "Migrating from 1.x" section.
  • END_OF_SPEECH is emitted per final transcript (rather than waiting for provider utterance-end events) deliberately: it preserves the low end-of-utterance latency the finalize flow provides.
  • Both synthesize() and the streaming path treat a server close after audio was received as end-of-segment deliberately: several bridge providers (Rime, Cartesia) terminate segments via close.
  • Several findings from this PR's automated review were confirmed and fixed in follow-up commits; each fix is covered by a test in our out-of-tree suite.

@metehan-slng
metehan-slng requested a review from a team as a code owner July 15, 2026 14:03
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@tinalenguyen tinalenguyen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hi, thanks for the PR! could you remove the test files to prevent noise?

devin-ai-integration[bot]

This comment was marked as resolved.

@metehan-slng

Copy link
Copy Markdown
Contributor Author

hi, thanks for the PR! could you remove the test files to prevent noise?

Done, removed the test files. Thanks!

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@tinalenguyen tinalenguyen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

few notes:

  • removing the default for STT might surprise users, are we able to restore that or make a note of it in the README for migration?
  • update_options dropped speed (tts.py:662) callers passing speed= now hit a TypeError, and speed is frozen at construction. Public API break not in the migration notes
  • control_profile captured but never sent (tts.py:407) — the constructor arg / TTSConnectionConfig field is only ever emitted as a log field; never a header, never in the init payload. A user setting it gets a silent no-op.
  • STT reconnect loop: no cap, backoff, or failover (stt.py:939) a graceful WS close with no buffered audio and input still open makes recv_task return False, which reconnects the same candidate with no attempt counter

Comment thread livekit-plugins/livekit-plugins-slng/livekit/plugins/slng/tts.py Outdated
Comment thread livekit-plugins/livekit-plugins-slng/livekit/plugins/slng/stt.py Outdated
…profile and prebuilt TTS candidates, cap silent STT reconnects, list-typed connections
@metehan-slng

metehan-slng commented Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

few notes:

  • removing the default for STT might surprise users, are we able to restore that or make a note of it in the README for migration?
  • update_options dropped speed (tts.py:662) callers passing speed= now hit a TypeError, and speed is frozen at construction. Public API break not in the migration notes
  • control_profile captured but never sent (tts.py:407) — the constructor arg / TTSConnectionConfig field is only ever emitted as a log field; never a header, never in the init payload. A user setting it gets a silent no-op.
  • STT reconnect loop: no cap, backoff, or failover (stt.py:939) a graceful WS close with no buffered audio and input still open makes recv_task return False, which reconnects the same candidate with no attempt counter

Thanks for the thorough review, all four addressed:

  • STT default: kept explicit (2.0 intentionally avoids defaulting to a specific provider) and added a migration bullet to the README.
  • speed in update_options: restored, including propagation to fallback candidates.
  • control_profile: removed entirely. It only ever affected direct-provider endpoints in our internal lineage; on Unmute Bridge endpoints it was already inert, and 2.0 is bridge-only, so it was a leftover with no possible effect. It never shipped in a released version, so no migration note.
  • STT reconnect loop: now capped. Consecutive graceful closes that produce no transcripts (3 max) raise and go through the normal failover/exhaustion path; transcript progress or a candidate switch resets the budget.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@tinalenguyen tinalenguyen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for iterating, tested it as well! this will be included in the next patch release, all plugins must be on the same version. the readme migration notes and deprecation warnings should suffice, we could also make note on our docs

@tinalenguyen
tinalenguyen merged commit 08cd26b into livekit:main Jul 23, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants